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Abstract 

Cooper-Harper  ratings  (CHRs)  have  been  used  to  describe  and  compare  aircraft 
handling  qualities  for  over  40  years,  but  are  by  their  very  nature,  subjective.  The 
subjective  and  sometimes  ambiguous  results  obtained  from  qualitative  handling  quality 
ratings  are  inconsistent  with  the  rest  of  the  flight  test  process,  where  quantifiable  results 
followed  by  statistical  analysis  are  the  nonn.  This  thesis  presents  a  method  for  obtaining 
accurate  and  consistent  flight  test  data  that  quantifies  the  handling  qualities  of  a  specific 
aircraft.  The  method  is  demonstrated  using  both  pilot-in-the-loop  simulations  and  flight 
tests  with  the  NF-16D  Variable-Stability  In-Flight  Simulator  Test  Aircraft  (VISTA). 

Boundary  Avoidance  Tracking  (BAT),  introduced  in  2004  by  Mr.  William  Gray 
III,  a  test  pilot  at  the  US  Air  Force  Test  Pilot  School  (TPS),  is  used  here  to  provide  a 
novel  approach  for  forcing  an  increase  in  pilot  workload  and  tracking  performance  in 
order  to  assess  an  aircraft’s  handling  qualities.  By  utilizing  BAT  with  shrinking  desired 
performance  boundaries  on  a  point  tracking  task,  pilots  are  forced  to  their  maximum 
performance  (i.e.  minimum  error)  on  the  tracking  task.  This  maximum  achievable  BAT 
performance  can  then  be  used  as  a  measure  of  the  aircraft’s  handling  “quality”. 

The  BAT  method  of  assessing  an  aircraft’s  handling  quality  was  used  with  both 
pilot-in-the-loop  simulations  and  flight  tests  to  obtain  quantitative  tracking  perfonnance 
data.  This  data  was  compared  and  correlated  to  CHR  data.  In  order  to  collect  the  data,  a 
6-degrees-of- freedom  (DOF)  pilot-in-the-loop  F-16  simulator  was  developed  and 
implemented  on  a  desktop  computer.  Twenty  seven  test  subjects  flew  the  BAT  profile  on 
the  desktop  simulator;  these  subjects  also  flew  the  same  profile  in  AFRL/VA’s  Infinity 


Cube  simulator.  Data  from  these  two  simulations  were  used  to  develop  a  flight  test  plan 
for  implementation  on  AFRL/VA’s  Large  Amplitude  Multi-mode  Aerospace  Research 
Simulator  (LAMARS)  and  on  TPS’s  NF-16D  VISTA.  Seven  test  subjects  then  flew  a 
modified  BAT  profile  on  the  desktop  simulator,  the  LAMARS,  and  13.7  flight  hours  in 
the  VISTA.  Data  collected  included  tracking  and  boundary  information,  as  well  as 
CHRs  for  each  of  four  different  pitch  control  models.  Results  supporting  the  existence  of 
boundary  awareness  were  found,  as  well  as  a  correlation  between  total  bounded 
simulation  run  time  and  Cooper-Harper  rating. 
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AN  ANALYSIS  OF  AIRCRAFT  HANDLING  QUALITY  DATA  OBTAINED 
FROM  BOUNDARY  AVOIDANCE  TRACKING  FLIGHT  TEST  TECHNIQUES 


1.  Introduction 


1.1  Motivation 

Accurate  and  consistent  data  that  quantify  the  handling  qualities  of  a  specific 
aircraft  are  difficult  to  acquire.  Cooper-Harper  ratings  (CHRs)  (Cooper,  1966)  have  been 
used  to  describe  and  compare  aircraft  handling  qualities  for  over  40  years,  but  are  by  their 
very  nature,  subjective.  Additionally,  the  data  obtained  through  Cooper-Harper  ratings 
are  difficult  to  reduce  (i.e.  you  can’t  average  CHRs),  and  by  assigning  a  single  CHR  to  an 
aircraft,  some  data  may  be  lost. 

Current  handling  qualities  flight  test  techniques  call  for  the  test  pilot  to  perform 
an  operationally  representative  task,  and  then  rate  the  aircraft  using  the  Cooper-Harper 
scale  (Appendix  E).  This  rating,  when  pooled  with  other  pilots’  ratings,  is  used  to 
categorize  the  aircraft’s  handling  qualities.  The  two  primary  considerations  of  the  pilot 
assigning  a  CHR  are  task  performance  and  pilot  workload.  If  a  pilot  performs  as  desired 
on  the  task,  but  is  working  extremely  hard,  then  the  aircraft  is  given  a  downgraded  rating. 
Similarly,  if  a  pilot  performs  poorly  on  the  task,  but  is  not  working  very  hard,  a 
downgraded  rating  is  also  given,  even  though  the  pilot  might  have  been  able  to  achieve 
better  perfonnance  with  a  higher  workload.  The  subjective  nature  and  variability  of  how 
a  pilot  defines  his  or  her  workload  may  greatly  influence  the  CHR.  Once  the  CHRs  from 
several  different  pilots  have  been  gathered,  there  is  no  definitive  guidance  on  interpreting 
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the  data.  An  informal  survey  of  flight  test  professionals  produced  several  different 
methods  for  interpreting  a  histogram  distribution  of  CHRs. 

The  subjective  and  sometimes  ambiguous  results  obtained  by  qualitative  handling 
quality  ratings  are  inconsistent  with  the  rest  of  the  flight  test  process,  where  quantifiable 
results  subjected  to  statistical  analysis  is  highly  desired. 


1.2  Research  Objectives  and  Hypotheses 

The  objective  of  this  research  was  to  determine  if  a  bounded  tracking  task  could 
be  used  to  produce  numerical  aircraft  handling  qualities  data.  To  do  this,  four  goals  were 
developed  that  directed  the  research  and  experimentation: 

1.2.1  Goal  1  -  Determine  if  boundaries  influence  task  performance. 

If  boundaries  can  affect  pilot  performance,  then  specifically  designed  boundaries 
might  be  used  to  “force”  the  pilot  to  alter  pilot  workload  and  achieve  different 
performance  levels. 

1.2.2  Goal  2  -  Determine  if  boundaries  can  increase  task  performance 

If  a  specific  sequence  or  type  of  boundary  can  increase  task  perfonnance,  then 
that  sequence  or  type  of  boundary  can  be  used  to  drive  the  pilot  to  the  best  perfonnance 
possible  in  that  aircraft  on  a  specific  task. 
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1.2.3  Goal  3  -  Determine  if  the  performance  on  bounded  tracking  tasks  can 
be  used  to  estimate  aircraft  handling  qualities. 

If  pilots  can  be  forced  to  perform  to  the  limit  of  their  abilities,  then  the  maximum 
achievable  task  perfonnance  (by  any  pilot)  can  be  obtained  in  an  aircraft  on  a  specific 
task.  This  performance  limit  can  be  used  as  a  determination  of  the  aircraft’s  best  possible 
performance,  or  its  handling  “quality”. 

1.2.4  Goal  4  -  Quantify  the  relationship  between  bounded  tracking 
performance  and  Cooper-Harper  ratings. 

Finding  a  correlation  between  BAT  data  and  CHRs  will  help  validate  the  use  of 
BAT  data  in  handling  qualities  flight  test,  and  possibly  focus  future  test  on  relevant 
avenues  of  research. 


1.3  Experiment  Overview 

An  F-16  aircraft  model  and  bounded  tracking  task  were  developed  on  a  desktop 
computer  and  27  test  subjects  flew  the  tracking  task  with  variable  boundaries  and  3 
different  stabilator  rate  limits  (60,  30  and  15  degrees  per  second).  The  27  test  subjects 
then  flew  a  similar  aircraft  model  and  identical  tracking  task,  with  the  same  boundaries 
and  rate  limits  on  the  Air  Force  Research  Laboratory’s  (AFRL)  Infinity  Cube  simulator. 
The  results  were  compared  to  determine  if  the  different  control  schemes  present  in  the 
two  simulators  produced  different  tracking  performance.  Data  collected  focused  on  the 
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tracking  error  as  a  measure  of  task  performance,  as  well  as  the  length  of  time  the 
simulation  was  flown  prior  to  impacting  a  boundary. 

Research  was  continued  as  part  of  the  USAF  TPS  Test  Management  Project 
(TMP)  BAT  DART,  at  Edwards  AFB,  CA.  Pitch  control  models  representative  of 
Cooper-Harper  levels  1,  2,  and  3  were  developed  and  flown,  in  addition  to  the  previously 
used  desktop  computer  pitch  control  model,  on  the  AFRL  Large  Amplitude  Multi-mode 
Aerospace  Research  Simulator  (LAMARS)  and  TPS’s  Variable-Stability  In-Flight 
Simulator  Test  Aircraft  (VISTA).  Cooper-Harper  perfonnance  rating  definitions  were 
developed  for  the  previously  mentioned  tracking  task,  and  an  additional  7  test  subjects 
flew  the  tracking  task  once  with  no  boundaries  to  produce  a  baseline  CHR,  and  once  with 
boundaries.  Data  collected  included  all  aircraft  performance  and  longitudinal  axis  data, 
as  well  as  Heads-Up  Display  (HUD)  and  Multi-Function  Display  (MFD)  video. 


1.4  Preview  of  Results 

The  initial  simulator  studies  conducted  at  the  Air  Force  Institute  of  Technology 
(AFIT)  were  focused  on  achieving  different  tracking  performance  on  one  aircraft  model 
with  different  stabilator  rate  limits.  Tighter  boundaries  produced  smaller  errors  in 
tracking  performance  up  to  a  point.  When  the  boundaries  became  too  small,  the  tracking 
errors  usually  increased  and  a  boundary  was  often  rapidly  impacted.  Despite  achieving 
the  goal  of  improving  tracking  performance  with  boundaries,  different  tracking 
performance  was  not  achieved  with  different  stabilator  rate  limits.  Pilot  performance 
with  15  degrees  per  second  stabilator  rate  limit  was  almost  the  same  as  the  model  with  60 
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degrees  per  second  stabilator  rate  limit.  However,  the  two  different  simulators  (Desktop 
simulation  and  Infinity  Cube  simulator)  produced  markedly  different  tracking 
performance.  The  difference  in  pitch  control  models  was  theorized  to  have  caused  the 
difference  in  perfonnance.  However,  additional  differences  in  the  simulation 
environment  and  hardware  could  have  caused  the  different  tracking  perfonnance,  and  the 
effects  of  these  (mostly)  unknown  variables  could  not  be  eliminated. 

The  three  different  pitch  control  models  that  were  later  developed,  in  addition  to 
the  original  desktop  simulation  pitch  control  model,  produced  dramatically  different 
tracking  performances  on  the  VISTA,  with  the  same  environment,  hardware,  and  aircraft 
model.  An  apparent  correlation  between  Boundary  Avoidance  Tracking  (BAT)  task 
performance  and  CHRs  was  found. 

Additionally,  most  test  subjects’  initial  average  tracking  performance  did  not 
appear  to  be  related  to  boundary  size.  Farther  into  a  given  model  run,  when  the 
boundaries  decreased  in  distance  from  the  tracking  target  to  a  certain  size,  nearly  all  test 
subjects  began  perfonning  better  on  the  tracking  task  at  the  same  boundary  value.  This 
point  is  theorized  to  be  the  point  of  “Boundary  Awareness”  and  appeared  to  be  consistent 
throughout  the  different  pitch  control  models. 


1.5  Thesis  Overview 

In  Chapter  2,  a  review  of  the  previous  work  conducted  by  Mr.  Gray  on 
Boundary  Avoidance  Theory  (BAT)  is  provided.  Additional  boundary  avoidance  work 
done  by  Capt  Randy  Warren  for  his  thesis  “An  Investigation  of  the  Effect  of  Boundary 
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Avoidance  on  Pilot  Tracking”  (Warren,  2006)  and  USAF  TPS  TMP  “HAVE  BAT” 
(Warren  et.  all,  2006)  is  also  addressed.  Chapter  3  presents  the  design  of  the  boundary 
avoidance  tracking  (BAT)  task  and  boundary  profile.  Chapter  4  details  the  research 
conducted  at  the  Air  Force  Institute  of  Technology,  to  include  the  Desktop  (DT) 
computer  simulator  and  AFRL’s  Infinity  Cube  simulator,  as  well  as  the  experimental 
setup  and  results  of  the  tracking  task.  Chapter  5  chronicles  the  research  conducted  at  TPS 
under  the  ‘BAD  DART”  TMP  and  the  results  of  the  LAMARS  and  VISTA  flight  test. 
Chapter  6  presents  the  conclusions  and  recommendations  as  well  as  a  summary  of  the 
results. 
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2.  Background 


2.1  Handling  Qualities  Testing 

Traditional  handling  qualities  testing  uses  tracking  tasks  to  simulate  an 
operational  representative  environment  or  task  (DoD,  1990).  For  example,  a  test  pilot 
might  be  instructed  to  track  another  aircraft  with  a  HUD  pipper,  which  is  very 
operationally  representative.  Following  the  task,  the  test  pilot  is  asked  to  rate  the  aircraft 
on  that  specific  task  using  CHRs  (Appendix  F).  This  task  is  a  good  example  of  point 
tracking. 

2.1.1  Point  Tracking 

Point  tracking  is  where  the  pilot  is  given  a  target,  or  “point”,  and  told  to  track  it. 
The  pilot’s  inputs  to  the  aircraft  control  system  are  intended  to  maintain  the  desired 
condition.  As  the  aircraft  or  system  deviates  from  the  desired  track  condition,  the  pilot 
alters  his  or  her  inputs  to  regain  the  track.  Generally,  as  the  tracking  error  grows,  the 
pilot  will  use  larger  control  inputs  to  correct  the  track  error.  Figure  1  illustrates  a  simple 
point  tracking  task. 


Figure  1  -  Example  Point  Tracking  Task 
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A  bicycle  rider  is  instructed  to  ride  down  a  road  and  “track”  the  centerline  of  the 
road.  Various  external  inputs  into  the  system  (rocks,  bumps  in  the  road,  winds,  etc. . .) 
may  cause  tracking  errors  that  should  be  corrected.  The  farther  away  from  the  centerline 
that  the  bicycle  rider  finds  himself,  the  larger  the  input  will  generally  be  to  correct  the 
error.  Assuming  that  fatigue,  a  learning  curve,  and  asymmetrical  external  inputs  into  the 
system  do  not  exist,  the  rider’s  average  absolute  track  error  will  be  consistent  from  one 
section  to  another  identical  section. 

2.1.2  Cooper-Harper  Ratings 

Prior  to  the  tracking  task,  the  rider  would  be  informed  that  the  desired 
performance  of  the  bicycle  is  to  keep  the  wheels  within  1  foot  of  the  centerline  for  the 
entire  ride.  Adequate  perfonnance  would  be  defined  as  keeping  the  wheels  within  3  feet 
of  the  centerline  for  the  entire  ride.  These  definitions,  in  addition  to  the  rider’s 
performance,  and  some  measure  of  the  workload  required  to  obtain  that  performance  on 
the  task,  are  what  is  required  to  generate  a  Cooper-Harper  rating.  Following  the  tracking 
task,  the  rider  would  be  asked  to  rate  the  bicycle  on  the  Cooper-Harper  rating  scale 
(Appendix  F),  using  performance  and  workload  to  determine  the  CHR.  We  can  assume 
that  the  rider’s  experience  level  will  have  some  effect  on  how  easily  he  or  she  can  track 
the  target.  A  6  year  old  who  just  learned  to  ride  a  bicycle  will  have  large  average  and 
peak  errors  compared  to  Lance  Armstrong,  as  an  extreme  example. 
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2.2  Boundary  Avoidance  Tracking  (BAT) 


Now  let  us  assume  that  there  are  real  physical  boundaries,  instead  of  desired 
performance  boundaries,  imposed  on  the  bicycle  rider.  We  will  suspend  the  6  foot  wide 
“road”  over  the  Grand  Canyon,  and  ask  the  rider  to  perform  the  same  task. 


Figure  2  -  Example  Bounded  Point  Tracking  Task 


The  tracking  task  and  target  have  not  changed,  and  the  boundaries  haven’t  really 
changed.  It  is  still  “adequate”  to  maintain  an  error  of  less  than  3  feet  from  the  centerline 
of  the  road.  Only  the  consequence  of  “impacting”  the  boundary  has  changed.  However, 
we  can  surmise  that  the  rider’s  control  inputs  will  change  when  a  boundary  is  approached 
compared  to  the  unbounded  case  shown  in  Figure  1 .  In  this  case,  as  a  boundary  is 
approached,  the  rider  will  change  his  or  her  inputs  to  avoid  the  boundary.  The  control 
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inputs  at  any  given  instant  are  theorized  to  be  approximately  proportional  to  the  time 
remaining  to  impact  the  boundary  at  that  instant  following  a  slight  lag  due  to  pilot 
perception  (Gray).  The  theorized  relationship  of  the  inputs  to  the  boundary  will  be 
examined  more  in  Section  2.3.1. 

Mr.  Gray,  an  instructor  at  the  USAF  Test  Pilot  School,  introduced  the  concept  of 
Boundary  Avoidance  Tracking  (BAT)  in  a  paper  to  the  Society  of  Experimental  Test 
Pilots  in  2004  (Gray,  2004).  Mr.  Gray  proposed  a  type  of  pilot-induced  oscillation  (PIO) 
that  was  driven  by  the  boundaries  on  a  pilot’s  point  tracking  task.  When  the  boundaries 
got  too  close  to  the  tracking  target,  pilots  would  cease  tracking  the  target  and  track  the 
boundaries  instead,  decreasing  their  point  tracking  perfonnance. 
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2.2.1  Desired  vs.  Achieved  Performance 


The  following  figure  is  taken  from  the  USAF  TPS  Flying  Qualities  curriculum 
(TPS,  2006). 
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Figure  3  -  Effect  of  Desired  Performance  on  Achieved  Tracking  Performance 
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Figure  3  illustrates  another  aspect  of  Mr.  Gray’s  theory:  that  increasing  the 
expectations  on  a  pilot’s  performance  (i.e.  narrowing  the  boundaries)  will  increase  the 
performance  achieved.  As  can  be  seen  from  the  figure,  there  is  a  theoretical  minimum  to 
the  achieved  tracking  performance,  or  minimum  error  achievable.  If  many  different 
pilots  flew  the  same  task  on  the  same  aircraft,  many  different  tracking  performance 
curves  could  be  plotted  on  the  same  chart.  Then,  the  curve(s)  with  the  lowest  achieved 
tracking  error  could  be  used  to  postulate  the  minimum  error  achievable  for  that  specific 
aircraft  on  that  specific  task. 

An  additional  possible  effect  of  adding  boundaries  to  a  tracking  task  can  be  seen 
in  the  red  line  portion  of  Figure  3.  When  the  boundaries  or  expectations  on  the  tracking 
task  become  too  restrictive,  the  pilot  will  cease  to  track  the  target  and  begin  tracking  or 
avoiding  the  boundaries.  The  pilot’s  perfonnance  will  suffer,  tracking  errors  will  grow, 
and  a  PIO  may  be  encountered. 

2.2.2  Testing  Limitations  on  Simulating  Real  Boundaries 

One  of  the  difficulties  in  using  boundaries  in  simulators  or  actual  flight  test  is 
simulating  a  real-world  boundary,  like  the  ground.  The  consequences  of  an  aircraft 
unintentionally  hitting  the  ground  are  almost  always  catastrophic.  Hence,  the  pilot 
response  generated  by  approaching  this  type  of  boundary  is  nearly  always  very  rapid  and 
of  large  magnitude  -  usually  at  the  maximum  capability  of  the  pilot  and  control 
inceptor(s).  It  is  this  maximum  deflection  input  and  aircraft  response  that  is  so  difficult 
to  reproduce  with  simulated  boundaries. 
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2.3  Previous  Testing 


2.3.1  Mr.  Gray’s  Desktop  Simulator 

Following  theoretical  and  computer  modeling  work  on  his  theory,  Mr.  Gray 
developed  a  pilot-in-the-loop  simulation  using  the  flying  qualities  data  of  a  North 
American  Navion  (Gray,  2005:6).  Using  Matlab©  and  Simulink©,  Mr.  Gray  presented  a 
bounded  tracking  task  to  the  subject  pilots.  The  tracking  task  consisted  of  matching  the 
altitude  of  a  “lead”  Navion  that  was 


constantly  changing  (Gray,  2005:6). 
The  test  subjects  were  provided 
with  a  display  that  showed  the 
relative  altitude  of  the  pilot’s 
aircraft  (also  a  Navion  model)  with 
respect  to  the  lead  Navion  (Figure 
4).  The  task  was  to  minimize  the 
altitude  error  between  the  two 
aircraft  by  maintaining  their  aircraft 
altitude  delta  line  on  the  zero  error 
reference  and  to  avoid  impacting 
the  boundaries. 
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Figure  4  -  Mr.  Gray’s  Navion  Simulator  Display 
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The  boundaries  were  moved  closer  to  the  zero-error  reference  by  25%  once  every 
60  seconds,  and  the  oscillatory  flight  path  of  the  lead  Navion  was  repeated  every  minute 
as  well.  Eight  test  subjects  each  flew  three  different  scenarios:  first,  the  standard  Navion 
tracking  task  as  described  above;  second,  a  300  millisecond  time  delay  was  added  to  the 
pilot’s  inputs;  and  third,  a  17  degree  per  second  rate  limit  was  placed  on  the  deflection  of 
the  horizontal  tail  of  the  pilot’s  Navion  (Gray,  2005:6).  The  simulation  was  tenninated 
when  the  altitude  error  between  the  lead  and  subject’s  aircraft  exceeded  one  of  the  current 
boundary  limits. 

Mr.  Gray’s  data  reduction  focused  on  characterizing  certain  boundary  tracking 
parameters  which  focused  on  the  time  to  impact  the  boundary.  However,  he  also  noted  a 
loose  grouping  of  the  successful  tracking  times  of  the  different  scenarios  (Figure  5) 
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Figure  5  -  Grouping  of  Navion  Simulator  Tracking  Times  (Gray,  2004) 

The  Boundary  Tracking  Parameter  axis  shown  in  Figure  5  is  a  separate  parameter 
under  investigation  by  Mr.  Gray  and  holds  no  relevance  to  this  thesis.  Discussions  with 
Mr.  Gray  about  desired  vs.  achieved  performance  and  the  grouping  of  successful  tracking 
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times  of  different  flight  control  scenarios  set  the  stage  for  the  research  that  was  conducted 
for  this  thesis. 

2.3.2  Capt  Randy  Warren’s  Thesis  &  HAVE  BAT  TMP 

Captain  Randy  Warren’s  TMP  (Warren,  et.  All,  2006)  and  thesis  (Warren,  2006) 
characterized  BAT  in  a  dynamic  flight  environment.  Repeated  BAT  events  were  flown 
in  a  T-38C  aircraft  and  analyzed.  His  research  focused  on  characterizing  initial 
parameters  for  BAT,  and  found  that  the  time  to  boundary  impact  where  pilots  go  to 
maximum  feedback  gain  (tmax)  was  independent  of  pilot,  maneuver,  or  flight  conditions 
(Warren,  2006),  but  may  be  dependent  on  other  unknown  factors.  Warren’s  thesis  also 
addressed  other  BAT  parameters,  but  their  discussion  is  not  relevant  to  the  focus  of  this 
thesis,  and  will  not  be  discussed  further.  It  should  be  noted  that  the  information  collected 
and  analyzed  on  the  HAVE  BAT  TMP  was  the  first  in-flight  data  to  show  that  Boundary 
Avoidance  Tracking  (BAT)  exists. 
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3.  Tracking  Task  and  Boundary  Profile 


3.1  BAT  Flight  Path  Angle  (y)  Tracking  Task 

The  tracking  task  was  designed  to  closely  mirror  Mr.  Gray’s  Navion  tracking  task 
described  in  Section  2.3.1.  However,  since  the  ultimate  goal  was  to  fly  the  tracking  task 
in  the  NF-16D  VISTA  aircraft,  flight  path  angle  (y)  was  chosen  as  the  target,  instead  of 
an  altitude  differential  as  in  Mr.  Gray’s  simulator.  The  aircraft  flight  path  angle  (y),  or 
velocity  vector,  is  already  calculated  and  displayed  in  all  modern  HUD  equipped  aircraft. 

A  simulator  arrangement  similar  to  Mr.  Gray’s,  with  a  target  aircraft  model 
generating  target  information,  and  a  separate  pilot-in-the-loop  model  being  flown  by  the 
test  subject,  was  chosen.  The  target  aircraft  control  inputs  were  the  gained  sum  of  three 
sine  waves.  The  frequency  and  amplitudes  of  the  inputs  are  shown  in  Table  1.  The  sum 
of  the  three  sine  waves  was  gained  by  a  factor  of  200  prior  to  being  input  into  the  target 
aircraft  pitch  control,  to  provide  appropriate  target  aircraft  control  input  magnitude. 


Table  1  -  Sine  Waves  Used  Target  Aircraft  Control  Input 


Input 

Frequency 

(rad/sec) 

Amplitude 

Phase 

Fast  Sine  wave 

0.21 

tt/180 

0 

Medium  Sine  wave 

0.42 

-ti/180 

n/3 

Slow  Sine  wave 

0.84 

tt/180 

71(2/3) 

The  target  flight  path  profile  that  resulted  from  the  sum-of-sines  input  is  shown  in 
Figure  6.  There  is  a  short  repeat  to  the  pattern  approximately  every  30  seconds.  This 
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repeat  was  designed  into  the  simulation  to  ensure  that  the  tracking  task  presented  the 
same  level  of  difficulty  in  subsequent  boundary  step  sizes. 


Figure  6  -  Desktop  Simulation  Target  Aircraft  Flight  Path  Angle  (y)  Profile 


3.2  Boundary  Profile 

The  target  aircraft’s  flight  path  angle  was  the  actual  target  for  all  of  the  tracking 
tasks.  The  deviation  from  this  target  was  given  boundaries,  and  the  test  subjects  were 
instructed  to  treat  the  displayed  boundaries  just  as  they  would  treat  the  ground  or  another 
physical  object  that  would  cause  catastrophic  damage  if  they  contacted  them  with  the 
FPM.  The  boundary  was  initially  set  at  a  value  of  ±10°,  and  then  the  boundaries  were 
stepped  closer  to  the  target  by  25%  every  60  seconds;  the  same  boundary  step  size  as  Mr. 
Gray’s  Navion  simulator.  A  time  history  of  the  boundary  values  for  an  entire  simulation 
run  is  shown  in  Figure  7.  This  figure  shows  the  relative  distance  (in  degrees  displayed  in 
the  HUD)  of  the  boundaries  from  the  target  flight  path  angle. 
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Boundary  vector 


Figure  7  -  Boundary  Profile 

If  the  flight  path  error  (y error),  or  deviation  of  the  ownship  flight  path  angle  (y)  from  the 
target  flight  path  angle  (ytgt),  exceeded  the  current  boundary  limit,  the  simulation  was 
terminated. 
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4.  Air  Force  Institute  of  Technology  (AFIT)  Research 


4.1  Test  Subjects 

Twenty  seven  test  subjects  consisting  of  a  mix  of  tighter  pilots,  heavy/transport 
pilots,  and  non-pilots  participated  in  the  research  by  flying  a  desktop  simulator  at  AFIT. 
Relevant  flight  time  of  the  test  subjects  is  presented  in  Appendix  D.  Test  subjects  were 
instructed  to  track  a  target  line  and  to  treat  displayed  boundaries  as  if  they  were  life  or 
death  boundaries.  Some  role-play  was  required  to  be  a  good  test  subject.  A  steak  dinner 
was  also  offered  to  the  test  subject  that  flew  the  longest  without  impacting  a  boundary. 

The  same  27  test  subjects  that  participated  in  the  AFIT  desktop  simulator  also 
flew  the  simulation  in  AFRL’s  Infinity  Cube  simulator.  A  random  selection  of  test 
subjects  flew  the  Infinity  Cube  prior  to  the  desktop  simulator  in  an  effort  to  remove  the 
learning  curve  factor  from  the  two  different  simulator  runs.  The  increased  “realism”  of 
the  Infinity  Cube  simulator  due  to  the  wide  horizontal  display  caused  a  few  of  the  non¬ 
pilots  to  state  that  they  felt  a  slight  motion  sickness  following  the  end  of  their  simulator 
runs.  However,  no  test  subject  reported  reduction  in  tracking  capability  due  to  the 
phenomenon. 


4.2  AFIT  Desktop  Pilot-In-The-Loop  Simulator 

The  desktop  simulator  was  constructed  on  a  Windows™  based  dual  Xenon 
processor  computer  with  1  GB  of  RAM.  Separate  video  cards  drove  the  simulation 
display  and  the  control  display.  The  simulation  was  constructed  in  MatLab©  and 
Simulink©  and  used  Aircraft  Visual  Display  Software  (A YDS,  2005)  for  the  visual 
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displays  and  control  stick  interface.  A  Microsoft™  Flightstick  Pro  force  feedback 
joystick  was  used  as  the  control  inceptor,  but  no  force  profde  was  programmed. 

4.2.1  Aircraft  Aerodynamic  Model 

The  aerodynamic  model  data  used  for  the  desktop  simulation  was  obtained  from  a 
NASA  Langley  wind  tunnel  study  of  a  subscale  F-16  (Nguyen,  1979)  and  provided  as 
part  of  the  AFIT  ‘MECH  628’  course  on  flight  controls.  The  limits  of  the  data  used  were 
-10°  to  +45°  alpha,  and  ±30°  beta.  The  data  were  collected  up  to  an  airspeed  of 
approximately  Mach  0.6.  The  leading  edge  flap  (LEF)  deflection  data  was  merged  into 
the  aerodynamic  data  matrix,  and  included  deflection  limits,  but  no  deflection  rate  limits 
(Nguyen,  1979).  The  NASA  data  also  included  a  model  of  the  F-16  afterburning 
turbofan  engine,  in  which  the  thrust  response  was  modeled  with  a  first-order  lag.  The  lag 
time  constant  was  a  function  of  the  actual  engine  power  level  and  the  commanded  power 
(Stevens,  2003). 

The  data  consisted  of  values  for  the  body-axes  dimensionless  aerodynamic 
coefficients  of  the  F-16  model  divided  into  separate  data  files.  The  aerodynamic  data 
files,  engine  model,  and  associated  subroutines  can  be  found  in  Appendix  A  and  B  of 
“ Aircraft  Control  and  Simulation ”  (Stevens,  2004)  and  will  not  be  reproduced  in  this 
document.  For  reference,  the  state-space  model  of  the  open-loop  (unaugmented) 
longitudinal  dynamics  at  300  KIAS  and  15,000  feet  is  given  in  Appendix  A  (Witte, 

2004).  The  poles  of  that  state  space  model  are  listed  in  Table  2  below  to  give  an  idea  of 
the  type  of  open-loop  response  that  can  be  expected  at  these  flight  conditions. 
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Table  2  -  Desktop  Simulator  Model  Open-Loop  Longitudinal  Dynamics 


Model 

Bare  Aircraft  Poles 

(Qsd 

Csp 

Desktop 

-1.5196  ±2.0071j 
-0.0165  ±0.0787j 

2.52 

0.60 

Some  changes  were  made  to  the  simulation  conditions  to  more  accurately  reflect 
an  operationally  relevant  flight  condition  and  aircraft  response.  The  reference  center  of 
gravity  (Xcg)  location  used  in  the  desktop  simulation  was:  Xcg  =  0.35c  ,  where  c  is  the 

mean  aerodynamic  chord.  Initial  conditions  of  400  KCAS  and  10000  feet  Mean  Sea 
Level  (MSL)  were  chosen.  Initial  equilibrium  positions  were  set  by  trimming  the  control 
surfaces  for  straight  and  level  flight  at  the  desired  airspeed  and  altitude  using  the 
‘trimmer.m’  file  supplied  with  the  F-16  aerodynamic  data  (Stevens,  2004). 

For  the  BAT  DART  TMP,  the  initial  conditions  were  changed  to  350  KCAS  and 
20000  feet  MSL  to  mirror  the  conditions  chosen  for  the  in-flight  testing  portion.  These 
conditions  were  changed  due  to  VISTA  Variable  Stability  System  (VSS)  limitations  and 
terrain  safety  margin  concerns. 

4.2.2  Pitch  Control  Model 

The  pitch  control  model  for  the  desktop  simulator  was  initially  intended  to  mirror 
the  VISTA  pitch  control  system;  however,  the  VISTA  control  system,  which  allows  the 
aircraft  to  simulate  other  aircraft,  is  quite  complex.  So  for  the  research  conducted  on  the 
desktop  simulator,  a  simpler  pitch  control  model  was  chosen.  The  Advanced  Fighter 
Technology  Integration  (AFTI)  F-16  was  a  joint  NASA  and  USAF  program  to  integrate 
and  demonstrate  new  aviation  technologies.  The  aircraft  demonstrated  extreme 
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maneuverability  including  flat  turns  and  selective  fuselage  pointing  using  forward- 
mounted  canards,  and  contained  a  triplex  digital  flight  control  computer  system  with 
custom  programming  (NASA,  2002).  The  digital  flight  controls  incorporated  a  back-up 
mode  that  could  be  accessed  by  flipping  a  hardware  cockpit  switch  that  would  lock  out 
the  custom  digital  control  computers  and  implement  a  simple  delta  pitch  rate  feedback 
control  loop  to  enable  the  pilot  to  safely  recover  and  land  the  aircraft  in  case  of  control 
computer  failure.  Figure  8  shows  the  AFTI  back-up  pitch  rate  feedback  transfer 
functions. 


AFTI  3 


Modified  AFTI 


Figure  8  -  AFTI  F-16  Back-Up  Pitch  Control  Feedback 


The  AFTI  back-up  pitch  control  laws  provided  a  simple  starting  point  for  the 
desktop  simulator.  Feedback  and  total  control  gains  were  modified  to  produce  an 
adequate  handling  aircraft.  The  resulting  desktop  simulator  pitch  control  model  is  shown 
below  in  Figure  9,  and  reproduced  in  Appendix  B. 
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The  basic  F-16  pitch  stick  dead-zone  (-1.75  to  1.75  lbs)  and  pitch  stick  command 
gradient  (Appendix  B)  were  inserted  into  the  stabilator  control  path,  as  well  as  the  basic 
F-16  stabilator  actuator  limits,  stabilator  actuator  rate  limits,  and  stabilator  control  path 
transport  delay.  These  parameters  were  obtained  from  the  VISTA  flight  control 
schematics  (General  Dynamics,  1989).  The  same  F-16  simulation  model  was  used  for 
the  desktop  simulator  studies  at  AFIT  and  TPS. 


4.2.3  Simulink©  Environment 

The  rest  of  the  Simulink©  simulation  environment  is  shown  below  in  Figure  10, 
and  reproduced  in  Appendix  B. 
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Figure  10  -  Desktop  Simulation  Simulink©  Environment 


The  aircraft  model  that  the  test  subject  flew  (Figure  9)  was  inside  the  “F-16  AC 
Dynamics”  block  of  Figure  10.  The  control  inputs  and  visuals  were  input  to  and 
extracted  from  the  Matlab©  environment  through  the  AVDS  program  block. 

The  target  aircraft  (F-16)  dynamics  and  control  were  inside  the  “Target  F-16  AC” 
block.  An  identical  aerodynamic  model,  pitch  control  model,  and  thrust  model  as  the  test 
subject’s  aircraft  were  used  for  the  target  aircraft.  Control  inputs  to  the  target  aircraft 
were  provided  as  a  sum  of  three  sine  waves.  The  frequency  and  amplitude  of  the  inputs 
were  designed  to  keep  the  target  aircraft’s  flight  path  response  (ytgt)  roughly  within  a  ±15 
degree  limit.  They  were  also  designed  to  provide  a  control  input  response  that  was  at 
most,  50%  of  the  control  system  inceptor  limits,  actuator  limits,  and  actuator  rate  limits. 
A  proportional  throttle  controller  was  used  to  maintain  the  target  aircraft’s  airspeed 
within  3%  of  the  initial  conditions;  however,  the  altitude  deviated  up  to  50%  from  the 
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initial  altitude  due  to  the  flight  maneuvers.  A  proportional  throttle  controller  was  also 
used  to  match  the  test  subject’s  throttle  command  to  the  target  aircraft’s  throttle 
command. 

The  simulation  visuals  were  constructed  through  AVDS  and  consisted  of  a  typical 
modem  fighter  aircraft  Heads-Up  Display  (HUD)  with  airspeed,  altitude,  and  attitude 
information.  The  desktop  simulation  display  is  shown  in  Figure  11. 


Figure  11  -  Desktop  Simulation  Visual  Display 


Additionally,  the  target  aircraft  flight  path  angle  (ytgt)  was  shown  as  a  dashed 
horizontal  line.  The  current  boundaries  were  shown  as  solid  horizontal  lines  above  and 
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below  the  target  line.  Ownship  flight  path  angle  (y)  was  displayed  as  a  flight  path  marker 
(FPM),  which  is  typical  of  modem  HUDs.  Out-the-window  terrain  and  visuals  were 
inherent  capabilities  of  the  AVDS  software.  Sufficient  terrain  detail  and  range  were 
chosen  to  provide  no  distractions  from  the  tracking  task  and  HUD  information,  but  to 
provide  a  visual  attitude  reference  similar  to  that  in  an  actual  aircraft.  Recorded 
parameter  sinks,  links,  and  Simulink©  blocks  are  not  shown  in  Figure  10  for  simplicity. 

A  list  of  recorded  parameters  for  the  desktop  simulation  can  be  found  in  Appendix  C. 

4.2.4  Results 

As  stated  in  Section  1 .4,  the  initial  desktop  simulation  focused  on  obtaining 
dissimilar  handling  qualities  through  different  stabilator  rate  limits  on  the  pitch  control 
model.  Figure  12  illustrates  one  method  of  determining  ‘performance’  on  a  bounded 
tracking  task.  The  task  is  flown  until  boundary  impact,  and  then  the  boundary  value  (in 
degrees)  at  impact  is  plotted  vs.  the  average  tracking  error  in  the  5  seconds  prior  to 
boundary  impact.  This  method  shows  total  performance  of  a  simulation  run  represented 
as  how  tight  a  boundary  was  achieved.  It  also  can  show  oscillatory  motion  or  poor 
tracking  performance  just  prior  to  a  boundary  impact,  as  this  will  produce  a  data  point 
that  is  closer  to  or  above  the  ‘desired  line’.  The  ‘desired  line’  is  a  line  that  represented 
achieved  performance  exactly  matching  desired  performance  -  in  this  case,  average  flight 
path  tracking  error  equaling  the  boundary  value. 
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Figure  12  -  Desktop  Simulator  Boundary  Impact  Analysis 


As  can  be  seen  in  Figure  12,  varying  the  stabilator  rate  limit  produced  very  little 
differentiation  in  performance.  These  results  and  those  of  the  Infinity  Cube  simulator 
prompted  the  abandonment  of  stabilator  rate  limiting  as  a  handling  quality  driver. 


4.3  Infinity  Cube 

The  Infinity  Cube  was  an  out-the-window  visual  system  that  surrounded  the  pilot 
with  four  displays  to  provide  a  continuous,  collimated,  200°  horizontal  by  120°  vertical 
field-of-view  (AFRL,  2000).  It  provided  a  45: 1  contrast  ratio  and  an  effective  resolution 
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of  6.5  arc  min/line  pair.  Images 
were  collimated  to  between  -0. 1 1 
and  0.0083  diopters  to  present  a 
focus  distance  near  infinity. 

Aircraft  models  were  driven  on  a 
Fedora  Core  PC.  The  control 
inceptor  used  was  a  fixed  position 
force  sensing  sidestick  similar  in 
characteristics  to  the  control  stick 
installed  in  an  early  production  F- 
16. 

4.3.1  Aircraft  Aerodynamic  and  Pitch  Control  Model 

The  aerodynamic  model  flown  in  the  Infinity  Cube  was  one  with  basic  F-16  flight 
characteristics  developed  for  previous  research  work  done  by  Capt  Hanley  (Hanley, 

2003)  and  Capt  Witte  (Witte,  2004).  Hanley  began  with  the  same  bare  airframe  F-16 
model  used  in  the  desktop  simulation  listed  in  Section  4.2.1 .  He  then  modified  the 
longitudinal  poles  to  create  4  different  dynamic  models.  Hanley  then  computed  the 
optimum  angle-of-attack  (a)  and  pitch-rate  (q)  feedback  gains  to  move  the  closed  loop 
poles  back  to  the  desired  locations.  These  feedback  gains  were  then  combined  with  the 
bare  aircraft  open-loop  dynamics  to  produce  a  new  state  space  model.  This  model  was 
implemented  directly  in  the  Infinity  Cube  simulator.  The  Hanley  model  “B”  was  chosen 


Figure  13  -  Infinity  Cube  Simulator 
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due  to  the  ease  of  implementation,  similarity  to  the  desktop  simulator  aero  model,  and 
similarity  to  the  actual  VISTA  (F-16)  dynamics.  For  reference,  the  state-space  model  of 
the  resulting  open-loop  longitudinal  dynamics  is  given  in  Appendix  A.  The  longitudinal 
poles  of  that  state-space  model  are  listed  below  in  Table  3  (Witte,  2004). 


Table  3  -  Infinity  Cube  Simulator  Model  Open-Loop  Longitudinal  Dynamics 


Model 

Bare  Aircraft 
Poles 

cosp 

Csp 

Kq 

Ka 

Aircraft  Poles  with 
Stability  Augmentation 

Infinity 

Cube 

-1.43  ±  1.85j 
-.017  ±  .074j 

2.34 

0.61 

0.156 

0.123 

-2.17  ±  2.22j 
-.017  ±  .070j 

The  initial  conditions  chosen  for  the  tracking  tasks  were  the  same  as  that  of  the 
desktop  simulator,  400  KIAS  and  10,000  feet  PA.  The  Infinity  Cube  model  was  trimmed 
in  a  similar  manner  to  the  desktop  simulation,  but  setup  and  simulation  control  were 
performed  by  an  AFRL  Engineer.  No  stick  shaping  was  used  and  the  stick  force  was 
converted  directly  to  an  elevator  deflection  command,  and  then  fed  to  a  first  order 
elevator  actuator  model.  This  was  chosen  due  to  the  ease  of  implementation,  similarity  of 
response  to  a  recent  F-16  flight  by  the  author,  and  because  the  feedback  loops  and  gains 
were  already  combined  into  the  modified  state-space  A  matrix. 

4.3.2  Simulation  Environment 

The  HUD  information  displayed  to  the  test  subjects  was  identical  to  that  displayed 
in  the  desktop  simulator,  and  shown  in  Figure  11.  Airspeed,  altitude,  attitude,  target  line, 
boundary  lines  and  velocity  vector  were  displayed  as  the  same  color  and  proportional  size 
as  the  desktop  simulator.  Out-the-window  terrain  and  environment  were  similar  but 
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generated  by  the  Infinity  Cube  terrain  computer  and  software,  and  were  again  chosen  to 
minimize  pilot  distraction  but  also  provide  a  visual  attitude  reference.  The  presence  of  a 
wide  (200°)  horizontal  visual  reference  allowed  peripheral  vision  to  aid  in  attitude 
recognition  and  maintenance.  A  list  of  recorded  parameters  for  the  Infinity  Cube 
simulation  can  be  found  in  Appendix  C. 

4.3.3  Flight  Path  Angle  (y)  Tracking  Task 

The  tracking  task  presented  to  the  test  subjects  was  identical  to  the  desktop 
simulator  tracking  task  (Figure  6).  A  60  Hz  time  history  vector  of  the  desktop  simulation 
target  aircraft’s  flight  path  angle  (ytarget)  was  used  as  the  input  data  for  the  target  line  in 
the  Infinity  Cube  simulation.  The  boundary  profile  was  also  identical  to  the  desktop 
simulation  boundaries. 

4.3.4  Results 

The  Infinity  Cube  simulation  trials  were  conducted  at  the  same  time  as  the  AFIT 
desktop  simulation  trail,  and  were  still  under  the  assumption  that  stabilator  rate  limiting 
would  result  in  significantly  different  handling  qualities.  Figure  14  shows  the  results  of 
the  Infinity  Cube  simulations.  Boundary  value  (in  degrees)  at  impact  is  plotted  vs.  the 
average  tracking  error  in  the  5  seconds  prior  to  boundary  impact. 
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Figure  14  -  Infinity  Cube  Simulator  Boundary  Impact  Analysis 

Figure  14  clearly  illustrates  the  fact  that  the  change  in  stabilator  rate  limit  had 
very  little  effect  on  the  handling  qualities  performance  of  the  Infinity  Cube  simulation. 
All  of  the  data  is  tightly  grouped  at  small  boundary  values  and  low  average  error  values. 
The  figure  below  shows  the  desktop  model  simulation  data  plotted  with  the  Infinity  Cube 
simulation  data. 
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Figure  15  -  Infinity  Cube  and  Desktop  Simulator  Boundary  Impact  Analysis 


The  difference  in  task  performance  between  the  two  simulators  is  readily  apparent 
from  Figure  15.  Every  Infinity  Cube  simulation  trial  achieved  lower  average  flight  path 
track  error  just  prior  to  boundary  impact,  and  the  data  for  the  Infinity  Cube  simulator  is 
grouped  much  tighter  than  that  for  the  desktop  simulator.  More  importantly,  the 
minimum  achievable  error  (and  boundary  size),  is  much  smaller  for  the  Infinity  Cube 
simulation.  However,  as  stated  in  Section  1 .4,  these  differences  were  not  necessarily  due 
to  the  differences  in  pitch  control  models.  They  could  have  been  due  to  discrepancies  in 
the  simulation  environment  and  hardware,  and  the  effects  of  these  (mostly)  unknown 
variables  could  not  be  eliminated. 

The  conclusions  drawn  from  Figure  15  were  the  nucleus  for  the  BAT  DART 
TMP,  where  4  different  pitch  control  models  were  flown  on  one  system,  with  identical 
hardware,  visual  systems,  control  inceptors,  and  in-flight  initial  conditions. 
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5.  USAF  TPS  Test  Management  Project  (TMP)  ‘BAT  DART’ 


5.1  Test  Subjects 

Seven  test  subjects,  all  TPS  students  or  faculty  and  members  of  the  BAT  DART 
team,  were  used  during  the  simulation  runs  at  TPS.  The  relevant  flight  experience  of  the 
test  subjects  is  presented  in  Appendix  D. 


5.2  Test  Objective 

The  BAT  DART  Test  Management  Project  (TMP)  was  conducted  under  the 
syllabus  of  the  USAF  Test  Pilot  School  (TPS)  as  part  of  the  joint  Air  Force  Institute  of 
Technology  (AFIT)  -  TPS  program  and  in  support  of  this  thesis  (Dotter  and  others, 

2006). 

The  objective  of  the  BAT  DART  test  program  was  to  detennine  if  the  pilot  plus 
aircraft  perfonnance  on  a  bounded  pitch  tracking  task  could  be  correlated  to  Cooper- 
Harper  ratings  for  longitudinal  handling  qualities,  and  to  detennine  that  correlation.  Four 
different  pitch-control  models,  including  one  model  designed  for  the  desktop  computer 
simulator,  were  flown  by  the  seven  BAT  DART  team  members  on  the  Large  Amplitude 
Multi-mode  Aerospace  Research  Simulator  (LAMARS)  and  on  the  NF-16D  Variable- 
Stability  In-Flight  Simulator  Test  Aircraft  (VISTA).  The  seven  BAT  DART  team 
members  also  flew  the  desktop  simulator,  with  initial  conditions  changed  to  match  the 
VISTA  and  LAMARS  flight  profiles. 

The  test  project  consisted  of  two  full  days  of  simulator  studies  in  the  LAMARS  at 
Wright  Patterson  AFB,  Ohio,  two  2-hour  ground  tests,  and  13.7  flight  hours  in  the 
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NF-16D  VISTA.  Flight  test  was  accomplished  from  8  to  18  September  2006.  Ten  test 
sorties  were  flown  in  NF-16D  86-00048  within  the  R-2508  complex  at  Edwards  Air 
Force  Base,  California. 

5.3  Flight  Path  Angle  (y)  Tracking  Task 

The  simulation  envelope  of  the  VISTA  and  terrain  located  in  the  R-2508  airspace 
utilized  at  TPS  necessitated  a  change  to  the  initial  flight  conditions  for  the  VISTA.  The 
desktop  simulator  and  LAMARS  initial  conditions  were  changed  to  match  those  chosen 
for  the  VISTA  to  facilitate  commonality  between  the  data  sets.  The  tracking  task 
presented  to  the  test  subjects  at  TPS  was  not  changed  from  that  used  at  AFIT;  however, 
because  the  initial  conditions  were  changed,  the  flight  path  response  from  the  same  inputs 
was  different.  The  ytgt  profde  produced  and  used  at  TPS  is  shown  in  Figure  16. 


Pitch  tracking  task  profile 


Figure  16  -  USAF  TPS  Desktop  Simulation  Target  Flight  Path  Profile 
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5.4  Cooper-Harper  rating  (CHR)  task 

The  desktop  simulator  runs  at  TPS,  as  well  as  the  LAMARS  and  VISTA,  also 
included  a  Cooper-Harper  rating  task.  This  tracking  task  consisted  of  the  same  target 
profile  as  the  boundary  task,  but  with  no  boundaries.  Also,  the  flight  path  marker  was 
changed  from  the  typical  HUD  “aircraft  like”  symbol  to  two  concentric  circles  with  radii 
of  15  and  25  mils  respectively.  The  test  subjects  were  instructed  to  track  the  target  line 
with  the  new  CHR  flight  path  marker  for  one  minute.  Desired  performance  was  defined 
as  maintaining  the  target  line  inside  the  inner  circle  for  90%  of  the  time.  Adequate 
performance  was  defined  as  maintaining  the  target  line  inside  the  outer  circle  for  90%  of 
the  time.  At  the  end  of  the  one  minute  run,  the  test  subject  was  asked  to  rate  the  aircraft 
on  the  Cooper-Harper  scale  using  perceived  performance  and  pilot  workload.  Figure  17 
shows  the  changes  for  the  CHR  task  to  the  desktop  simulation  HUD. 


Figure  17  -  TPS  Desktop  Simulation  HUD  -  CHR  task 
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5.5  TPS  Desktop  Simulator 

All  model  runs  conducted  at  TPS  used  a  60  deg/sec  stabilator  rate  limit.  The  TPS 
portion  of  the  desktop  simulator  study  was  conducted  in  an  attempt  to  ensure  continuity 
of  data  with  the  LAMARS  and  the  VISTA  data  sets.  However,  the  differences  in 
simulator  hardware,  control  inceptors,  and  software  interfaces  negated  any  value  in 
comparing  the  results  in  the  manner  anticipated.  The  results  of  the  TPS  portion  of  the 
desktop  simulator  study  are  shown  below  in  Figure  18. 
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Figure  18  -  Comparison  of  AFIT  and  TPS  Desktop  Simulation  Results 
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5.6  LAMARS 


The  Large  Amplitude  Multi-Model  Aerospace  Research  Simulator  (LAMARS) 
was  a  5  degree-of-freedom  motion  based  simulator  operated  by  the  Air  Force  Research 
Laboratory.  The  LAMARS  consisted  of  a  simulator  dome  and  cockpit  mounted  on  the 
end  of  a  30  foot  arm.  The  simulator  dome  and  cockpit  could  rotate  ±25°  in  pitch,  roll  and 
yaw  at  a  maximum  of  60°  per  second.  The  arm  could  articulate  ±10  feet  vertically  or 
horizontally,  achieving  a  maximum  acceleration  of  ±3  g  vertically  and  ±2  g  horizontally. 


Figure  19  -  LAMAR  Simulator 

The  interior  of  the  LAMARS  display  dome  allowed  a  ±133°  horizontal  by  -20°  to 
±106°  vertical  image  composition.  Current  hardware  installed  in  the  display  dome  could 
project  a  ±60°  horizontal  by  ±20°  vertical  image  onto  the  interior  surface.  The  control 
inceptor  used  was  a  fixed  position  force  sensing  sidestick  similar  in  characteristics  to  the 
control  stick  installed  in  the  Infinity  Cube  and  early  production  F-16s.  The  LAMARS 
was  controlled  by  the  same  computers  used  for  the  Infinity  Cube  described  in  Section  4.3, 
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but  with  additional  motion  control  computers,  hydraulics,  and  different  visual  terrain 
projectors  and  software. 

The  primary  goal  for  the  LAMARS  portion  of  the  study  was  to  practice  flight  test 
techniques  developed  for  the  VISTA  in-flight  portion  of  the  BAT  DART  TMP,  as  well  as 
validate  the  difference  in  handling  qualities  that  should  be  present  in  the  4  different  pitch 
control  models  developed  and  selected  for  flight  test. 

5.6.1  Aircraft  Aerodynamic  and  Pitch  Control  Model 

The  initial  aerodynamic  model  flown  in  the  LAMARS  was  the  same  model  used 
in  the  Infinity  Cube.  This  state-space  model  was  re-trimmed  at  the  new  initial  conditions 
(350  KIAS  and  20,000  feet  PA),  wrapped  with  the  desktop  simulation  pitch  rate  feedback 
and  stick  shaping  characteristics,  and  became  the  DT  (desktop)  model.  In  addition  to  the 
previously  designed  desktop  simulation  pitch  control  model,  Calspan  Corporation  was 
contracted  to  design  3  different  pitch  control  models.  The  modes  were  required  to  be 
representative  of  a  Cooper-Harper  level  1,  level  2,  and  level  3  aircraft  in  the  longitudinal 
axis.  The  models  constructed  each  consisted  of  6  different  longitudinal  state-space  A 
matrices  that  corresponded  with  different  fuel  weights  anticipated  in  the  VISTA.  As  the 
VISTA  burned  fuel  in  flight,  different  fuel  weight  versions  of  the  same  level  model 
would  be  used  to  more  accurately  reproduce  the  same  handling  characteristics.  Only  the 
6,000  lb  fuel  weight  matrices  were  implemented  in  the  LAMARS  to  prevent  confusion 
and  simplify  testing  procedures.  These  fuel  weight  state-space  A  matrices  are  shown  in 
Appendix  F  along  with  the  B,  C,  and  D  matrices  common  to  all  models.  The  Calspan 
models’  open-loop  short  period  dynamic  characteristics  are  listed  below  in  Table  4.  The 
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basic  F-16  control  laws  utilized  previously  in  the  Infinity  Cube  were  used  for  roll  and 
yaw  control  of  all  models. 


Table  4  -  Calspan  Model  Open-Loop  Short  Period  Characteristics 


Model 

Short  Period  Poles 

Wsd 

Csp 

Level  1 

-3.15  ±  3.2137J 

4.5 

0.7 

Level  2 

-0.72  ±  2.2985j 

2.4 

0.3 

Level  3 

-0.31  ±  1.5187J 

1.55 

0.2 

Because  the  designed  control  feedbacks  were  inside  the  modified  state-space 
matrices,  the  control  stick  to  stabilator  path  was  implemented  as  a  direct  force  to  pitch 
rate  command. 

5.6.2  Simulation  Environment 

The  HUD  information  displayed  to  the  test  subjects  was  identical  to  that  shown  in 
Figures  1 1  and  17;  the  same  code  that  drove  the  Infinity  Cube  simulator  HUD  was  used 
to  drive  the  HUD  information  in  the  LAMARS.  Airspeed,  altitude,  attitude,  target  line, 
boundary  lines  and  velocity  vector  were  displayed  as  the  same  color  and  proportional  size 
as  the  desktop  simulator,  and  the  Infinity  Cube.  Out-the-window  terrain  and  environment 
were  similar  but  generated  by  a  separate  LAMARS  terrain  computer  and  software,  and 
were  again  chosen  to  minimize  pilot  distraction  but  also  provide  a  visual  attitude 
reference.  The  presence  of  a  wide  (120°)  horizontal  visual  reference  again  allowed 
peripheral  vision  to  aid  in  attitude  recognition  and  maintenance.  A  list  of  recorded 
parameters  for  the  LAMARS  can  be  found  in  Appendix  C. 
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5.6.3  LAMARS  Calibration 


After  some  pilot-in-the-loop  test  flying,  adjustments  were  made  to  the  pitch  stick 
gains  to  obtain  the  desired  pitch  response.  This  was  done  in  an  attempt  to  produce 
models  that  fell  into  the  target  CHR  range  of  level  1,  level  2,  and  level  3.  The  final  pitch 
stick  gains  are  shown  in  Table  5.  No  adjustment  was  made  to  the  pitch  stick  gain  for  the 
desktop  simulator  pitch  control  model;  it  was  implemented  as  shown  in  Figure  9. 


Table  5  -  Pitch  Stick  Gain  Changes  in  the  LAMARS 


Pitch  Control  Model 

Initial  Stick  Gain 

Final  Stick  Gain 

Level  1 

1.0 

1.00 

Level  2 

1.0 

0.25 

Level  3 

1.0 

0.50 

Desktop  Sim 

1.0 

1.00 

The  final  combinations  of  stick  gain  and  pitch  control  model  were  subjected  to  a 
10-lb  aft  stick  step  and  the  pitch  rate  was  recorded  for  comparison  and  verification  of  the 
VISTA  configurations.  The  results  of  the  step  inputs  are  shown  in  Figure  20.  The  level  3 
model  was  only  subjected  to  a  2  lb  aft  stick  step  input,  as  this  model  was  much  more 
sensitive  than  the  other  3  models.  The  final  stick  gains  shown  in  Table  5  were  used  as 
the  starting  point  for  the  VISTA  calibration  flight. 
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Time  (s) 


Figure  20  -  LAMARS  Models’  pitch  rate  response  to  step  input 


5.6.4  Test  Procedures 

The  same  seven  test  subjects  that  participated  in  the  desktop  simulator  study  at 
TPS  flew  the  LAMARS.  They  were  each  assigned  the  four  pitch  control  models  (Level 
1, 2,  3,  and  Desktop)  in  a  random  order.  The  test  subjects  were  allowed  one  minute  of 
free  flight  with  each  model  to  allow  adjustment  to  the  new  control  laws.  Each  subject 
then  flew  the  Cooper-Harper  rating  task  described  in  Section  5.4,  followed  by  the  BAT 
task  described  in  Section  3.1.  Data  were  collected  and  pilot  comments  were  recorded, 
and  then  the  subject  moved  on  to  the  next  pitch  control  model. 
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5.6.5  Results 


The  total  time  that  a  test  subject  flew  the  LAMARS  prior  to  a  boundary  impact 
was  recorded  for  each  pilot  on  each  model. 


Table  6  -  LAMARS  BAT  Boundary  Impact  Times 


Run  time  (seconds)until  boundary  impact 

Level  1 

Level  2 

Level  3 

DeskTop 

model 

model 

model 

model 

Pilot  1 

600 

458 

550 

549 

Pilot  2 

580 

507 

517 

521 

Pilot  3 

600 

489 

430 

518 

Pilot  4 

600 

546 

487 

486 

Pilot  5 

468 

489 

369 

416 

Pilot  6 

458 

437 

428 

428 

Pilot  7 

467 

490 

224 

368 

Average: 

539 

488 

469 

stdev: 

70.3 

34.7 

109.1 

66.4 

It  can  be  seen  in  Table  6  that  the  different  pitch  control  models  produced  different 
average  run  times.  Also  the  Level  1  model  produced,  on  average,  better  times  than  the 
Level  2  model,  which  produced  better  average  times  than  the  Level  3  model.  The 
Cooper-Harper  ratings  obtained  in  the  LAMARS  are  shown  below  in  Figure  2 1 . 
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LAMARS  Cooper-Harper  Rating  Histograms 


#  of  ratings 


Figure  21  -  LAMARS  Cooper-Harper  Ratings 


The  CHRs  show  decent  grouping  with  some  overlap  between  the  Level  1 , 2,  and  3 
models,  and  the  DT  model  CHRs  shows  a  very  strong  central  tendency.  A  correlation 
was  attempted  between  the  BAT  run  times  and  the  CHRs  obtained  in  the  LAMARS. 
Figure  22  below  shows  that  a  very  loose  correlation  may  be  seen,  but  the  data  is 
extremely  noisy  with  wide  confidence  intervals. 
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Simulation  Run  Time  Until  Boundary  Impact  (sec) 


Figure  22  -  LAMARS  CHR-BAT  Correlation 


The  data  was  further  reduced  in  an  effort  to  clean  up  the  correlation  by  removing  all  of 
the  non-pilots  from  the  data  set.  The  results  are  shown  below  in  Figure  23,  and  it  can  be 
seen  that  the  correlation  is  still  very  poor 
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Figure  23  -  LAMARS  CHR-BAT  Correlation  -  Only  Experienced  Pilots 


Taken  as  a  whole,  the  LAMARS  data  was  discouraging,  as  no  good  correlation 
was  found  and  the  boundary  impact  times  were  not  grouped  very  well  into  distinct  data 
sets  for  each  model.  However,  the  primary  LAMARS  goals  of  test  procedure  checkout, 
flight  test  technique  validation,  and  test  team  practice  were  met.  The  value  of  the 
LAMARS  and  other  simulator  data  will  be  further  addressed  in  Sections  6.1  and  6.2. 
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5.7  VISTA 


The  NF-16D  VISTA  was  a  unique  aircraft  based  on  a  Block  30  F-16D  in  the 
Peace  Marble  II  configuration.  The  avionics  were  Block  40  configuration  and  a  custom 
Digital  Flight  Control  Computer  (DFLCC)  was  installed.  The  Variable  Stability  System 
(VSS),  a  five  degree-of-freedom  simulator,  allowed  the  manipulation  of  flight  control 
parameters  in  order  to  simulate  specific  characteristics  of  many  different  flight  control 
systems  and  aircraft.  The  evaluation  pilot  (EP)  sat  in  the  front  cockpit  (FCP)  while  the 
safety  pilot  (SP)  controlled  the  VSS  from  the  rear  cockpit  (RCP). 


Figure  24  -  Variable-Stability  In-Flight  Simulator  Test  Aircraft  (VISTA) 


When  the  VSS  was  engaged,  the  pilot  controlled  the  aircraft  in  the  FCP  through  a 
sidestick  using  control  models  programmed  in  the  VSS.  Additional  control  modes 
allowed  both  the  EP  and  the  SP  to  fly  the  aircraft  through  the  sidestick  in  either  the  FCP 
or  RCP  with  a  basic  F-16  control  model.  The  displays  in  the  VISTA,  to  include  the 
HUD,  were  fully  configurable  and  re-programmable.  The  VISTA  HUD  was  programmed 
to  show  the  same  CHR  task  and  BAT  task  symbols  as  the  desktop  simulator  and  the 
LAMARS.  However,  due  to  pre-configured  portions  of  the  programmable  HUD,  the 
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altitude,  airspeed,  and  horizon  line  were  displayed  differently  than  those  of  the  other 
simulators.  This  can  be  seen  in  Figures  11,  17,  25,  and  28. 


Figure  25  -  VISTA  CHR  Task  HUD  View 


The  modified  altitude  and  airspeed  displays  were  determined  to  have  no  effect  on 
tracking  performance,  because  these  parameters  were  not  in  the  test  subjects  cross  check 
while  perfonning  the  tasks.  The  change  in  horizon  line  was  detennined  to  assist  the  test 
subjects  in  discriminating  between  the  target  line,  boundary  lines,  and  the  horizon  line, 
because  the  new  horizon  line  was  considerably  wider  than  the  target  and  boundary  lines. 

The  VISTA  was  capable  of  recording  HUD  and  Multi  Function  Display  (MFD) 
video  as  well  as  most  of  the  parameters  on  the  data  bus  of  the  VSS.  A  list  of  the  relevant 
data  parameters  recorded  for  the  BAT  DART  TMP  is  listed  in  Appendix  C. 
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Specific  modifications  to  the  NF-16D  VSS  software  for  the  BAT  DART  project  included: 


1)  Programming  to  simulate  four  different  pitch  control  models. 

2)  Programming  to  input  a  target,  boundary  profile,  and  CHR  pipper  into  the 
HUD,  as  shown  in  Figures  25  and  28. 

3)  Programming  to  provide  for  simulation  control  by  the  SP. 


5.7.1  Aircraft  Aerodynamic  and  Pitch  Control  Model 

The  4  pitch  control  models  and  pitch  stick  control  paths  were  implemented  on  the 
VISTA  in  the  same  manner  as  on  the  LAMARS.  All  six  state  space  matrices  for  each 
model  were  available  for  execution,  and  the  current  fuel  weight  was  used  to  select  the 
appropriate  state  space  model  at  the  beginning  of  each  model  run. 

5.7.2  Ground  Test  Procedures 

Two  2-hour  ground  tests  were  conducted  on  3 1  August  and  7  September  2006  to 
test  software  integration  on  the  VISTA.  Ground  electrical  and  hydraulic  power  was 
applied  to  the  VISTA,  and  a  control  computer  was  attached  which  allowed  test  inputs  and 
modifications  to  be  input  into  the  VSS.  In  this  ground  mode,  the  VISTA  acted  like  a 
simulator,  using  the  VSS  computers  and  sample  atmospheric  data  to  simulate  flight. 
Activation,  control,  and  termination  of  the  simulation  profiles  as  well  as  data  recording 
were  tested.  In-flight  procedures  were  also  tested  and  the  flight  cards  developed  for  the 
actual  test  were  used  to  simulate  and  practice  for  the  flight  test  missions. 
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5.7.3  VISTA  In-Flight  Calibration 

The  first  flight  of  the  test  program  was  used  to  calibrate,  verify  and  validate  the 
four  pitch  control  model  configurations,  as  well  as  practice  flight  test  techniques  and  data 
recording.  The  flight  test  calibrations  were  performed  at  the  same  altitude  and  airspeed 
planned  for  the  data  flights:  350  KIAS  and  20,000  feet  PA.  Inputs  used  included  stick 
raps  and  step  inputs  manually  applied  by  the  pilot,  and  automated  step  inputs  applied  by 
the  VS  S.  The  data  generated  by  these  maneuvers  were  recorded  by  the  VS  S. 

During  the  calibration  flight,  it  was  determined  that  the  stick  gains  were  too  high 
in  all  configurations.  Test  profiles  and  step  inputs  results  were  compared  to  the  results 
from  the  LAMARS  test  during  post-flight  analysis,  and  an  initial  approximation  was 
made  to  decrease  all  of  the  stick  gains  by  50%.  The  final  stick  gains  for  the  VISTA  pitch 
control  models  were  further  reduced  by  another  20%  on  the  first  data  flight,  and  were 
established  more  by  test  pilot  feel  than  by  analytical  comparison  to  the  LAMARS  pitch 
response.  Those  corrections  are  shown  below  in  Table  7. 


Table  7  -  VISTA  Pitch  Stick  Gain  Corrections 


Pitch  Model 

Initial  Stick  Gain 
(from  LAMARS) 

Total  Stick  Gain 

Correction 

Final  Stick  Gain 

Level  1 

1.00 

40% 

0.40 

Level  2 

0.25 

40% 

0.10 

Level  3 

0.50 

40% 

0.20 

Desktop 

1.00 

40% 

0.40 

A  step  response  comparison  (Figure  26)  of  the  final  LAMARS  and  VISTA  Level  1  stick 


gain-model  combinations  shows  a  large  discrepancy  in  the  measured  pitch  rate  response. 


However,  in  the  opinion  of  the  test  pilot  present  on  the  calibration  flight,  and  the  first  data 
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collection  flight,  the  final  stick  gains  chosen  for  the  VISTA  produced  responses  that  were 
within  10%  of  the  LAMARS  models’  responses.  The  discrepancies  in  the  pitch  rate 
response  to  stick  input  could  be  due  to  inaccuracies  in  the  LAMARS  models,  delays 
inherent  in  the  LAMARS  or  VISTA  pitch  stick  to  actuator  control  paths,  or  other 
unknown  factors. 


Figure  26  -  Comparison  of  LAMARS  and  VISTA  Level  1  Model  Pitch  Rate 
Response  to  a  10  lb  Aft  Stick  Step  Input. 

Historically,  this  is  not  surprising.  The  first  flight  of  the  YF-16  (Smith,  1979)  and 
fourth  flight  of  the  C-17  (Kendall,  1996)  both  encountered  severe  pilot  induced 
oscillations  (PIOs)  due  to  excessively  high  stick  response  sensitivity.  After  an  extensive 
series  of  flight  tests  to  optimize  the  small-displacement  sidearm  controller  in  the  F-16,  the 
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final  roll  control  gradient  was  only  about  one-fourth  the  value  originally  set  in  the 
simulator  (DoD,  1995).  Mitchell  and  Klyde  (Mitchell,  2005)  also  noted  in  their  paper  on 
testing  for  PIOs  that  in  general,  simulators  are  notoriously  bad  at  accurately  predicting 
actual  aircraft  responses. 

5.7.4  Flight  Test  Procedures 

Following  the  calibration  flight,  data  for  the  BAT  DART  investigation  were 
collected  on  the  remaining  nine  flights.  Prior  to  each  test,  the  aircraft  was  trimmed  in 
straight  and  level,  unaccelerated  flight  at  the  initial  conditions.  The  new  pitch  control 
model  was  initialized  and  the  SP  gave  the  EP  control  of  the  aircraft.  The  EP  was  allowed 
1  minute  of  free-flight  to  adjust  to  the  new  pitch  control  model.  After  free-flight  was 
tenninated,  the  SP  prepared  the  CHR  task. 

Setup  parameters  for  the  Cooper  Harper  task  were  20,000  ±  500  feet  pressure 
altitude  and  350  ±  10  KIAS.  The  Cooper-Harper  pitch- tracking  task  was  flown  for  one 
minute.  The  pilot  attempted  to  keep  the  HUD-displayed  CHR  pipper  (Figure  25)  over  the 
flight  path  angle  target  (ytarget)  for  as  long  as  possible,  making  as  aggressive  inputs  as 
necessary  in  an  effort  to  achieve  desired  performance.  Desired  performance  was  defined 
as  maintaining  the  target  line  inside  the  inner  circle  90  percent  of  the  time;  adequate 
performance  was  defined  as  maintaining  the  target  line  inside  the  outer  circle  90  percent 
of  the  time. 

The  tracking  task  (Figure  27)  presented  to  the  test  subjects  was  identical  to  the 
desktop  simulator  tracking  task  used  at  TPS  (Figure  16),  as  modified  from  the  original 
AFIT  desktop  simulation  tracking  task.  A  25  Hz  time  history  vector  of  the  TPS  desktop 
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simulation  target  aircraft’s  flight  path  angle  (ytarget)  was  used  as  the  input  data  for  the 
target  line  in  the  VISTA. 


Pitch  tracking  task  profile 


Figure  27  -  VISTA  Target  Flight  Path  Angle  (ytarget)  Tracking  Task  Profile 

The  rear  cockpit  safety  pilot  (SP)  controlled  the  throttle  to  maintain  350  ±  50 
KIAS  throughout  the  task.  After  one  minute  of  tracking,  the  task  was  terminated.  The 
EP  assigned  a  Cooper-Harper  rating  based  on  perceived  performance  and  workload.  If  a 
PIO  was  encountered,  the  pilot  assigned  a  PIO  rating  (appendix  E).  Any  comments  given 
by  the  EP  were  also  recorded. 

Following  the  completion  of  the  CHR  tracking  task,  the  EP  then  flew  a  boundary 
avoidance  tracking  (BAT)  task  using  the  same  flight  path  angle  target  (ytarget)  tracking 
profde  (Figure  27)  with  the  standard  FPM  (Figure  28)  instead  of  the  CHR  pipper.  Setup 
parameters  for  the  pitch  tracking  task  were  20,000  ±  500  feet  pressure  altitude  and  350  ± 
10  KIAS.  The  pilot  attempted  to  keep  the  flight  path  marker  (FPM)  over  the  moving 
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target  line  while  remaining  inside  the  displayed  boundaries.  The  SP  controlled  the 
throttle  to  maintain  350  ±  50  KIAS.  As  each  BAT  task  progressed,  the  boundaries 
tightened  in  25%  increments,  identical  to  the  boundary  profiles  used  on  the  desktop 
simulator.  Infinity  Cube  simulator,  and  LAMARS.  The  tracking  profile  tenninated  when 
the  center  of  the  FPM  crossed  either  boundary  line. 


Figure  28  -  VISTA  BAT  Task  HUD  View 

Successful  testing  required  diligent  role-playing  -  the  pilot  was  instructed  to  treat 
the  boundaries  as  real  threats,  like  the  ground  or  another  aircraft,  and  not  accept  an 
impact  if  at  all  possible.  However,  since  test  safety  was  of  primary  concern,  the  pilots 
could  limit  their  inputs  with  discretion  to  avoid  over-g,  excessive  buffet,  wing  rock,  or 
other  real-world  limits  such  as  Instrument  Meteorological  Conditions  (IMC).  The  pilot 
commented  when  these  real-world  limits  affected  decision  making  and/or  tracking. 
Successfully  tracking  the  target  required  significant  negative  and  positive  g  loading.  All 
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test  subjects  were  seated  in  the  front  seat  (EP  cockpit)  of  the  VISTA,  as  the  Safety  Pilot 
(SP)  retained  all  controls  required  for  start/taxi/takeoff/landing/shutdown/and  any 
contingences  in  the  rear  cockpit  (RCP). 

5.7.5  VISTA  Results  and  Analysis 

Two  primary  methods  were  developed  to  analyze  the  data  acquired  from  flight 
test.  These  methods  included  distinguishing  the  levels  of  aircraft  performance  based  on 
the  length  of  time  before  boundary  impact,  and  mean  tracking  error  from  the  target 
compared  to  boundary  size. 

5.7.5.1  Boundary  Impact  Time  Analysis 

The  first  method  of  detennining  aircraft  handling  qualities  was  based  on  length  of 
time  progressed  through  the  tracking  exercise  before  contacting  a  boundary.  The 
drawbacks  of  this  method  included  the  potential  for  small  pilot  mistakes  such  as 
momentary  inattention,  mannequin  effect  where  aircraft  motion  induces  pilot  arm/hand 
motion  on  control  inceptors,  or  external  factors  such  as  traffic  or  airspace  boundaries  to 
tenninate  a  run  early.  When  an  external  effect  caused  the  test  subject  to  terminate  the 
exercise  early,  they  were  instructed  to  start  the  tracking  task  completely  over  or  to  restart 
the  tracking  task  at  an  intermediate  boundary  value. 

As  can  be  seen  below  in  Table  8,  the  different  pitch  control  models  produced  very 
dissimilar  results  in  the  boundary  tracking  task.  There  was  some  overlap  in  both  Level  1 
compared  to  Level  2  results,  and  Level  2  compared  to  Level  3  results.  However,  there 
was  no  overlap  in  boundary  impact  times  between  Level  1  and  Level  3. 
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Table  8  -  VISTA  Boundary  Tracking  Task  Times 


Run  time  (seconds)until  boundary  impact 

Level  1 

Level  2 

Level  3 

DeskTop 

model 

model 

model 

model 

Pilot  1 

517 

459 

* 

548 

Pilot  2 

499 

339 

369 

488 

Pilot  3 

460 

457 

309 

499 

Pilot  4 

438 

367 

308 

546 

Pilot  5 

427 

338 

101 

428 

Pilot  6 

488 

427 

310 

371 

Pilot  7 

427 

220 

11 

457 

Average: 

465 

373 

477 

stdev: 

36.6 

84.9 

143.1 

63.8 

*  -  Incomplete  test  point  due  to  fuel  state 


A  one-tailed  t-Test  analysis  (appendix  F)  of  the  times  and  averages  of  the  Level  1, 
2,  and  3  results  showed  a  greater  than  99.99  percent  chance  that  the  results  are  from  3 
separate  models.  This  is  significant  because  it  shows  that  the  BAT  method  can 
differentiate  between  the  in-flight  handling  qualities  present  in  different  aircraft  models. 
These  results  appear  to  correlate  with  the  Cooper-ffarper  ratings  (CFIR)  collected  from 
each  pilot  on  all  pitch  control  models.  The  CHR  results  are  shown  below  in  Figure  29. 
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#  of  ratings 


Figure  29  -  VISTA  Cooper-Harper  Ratings 

A  one-tailed  t-Test  analysis  (Appendix  F)  of  the  CHR  of  the  Level  1,  2,  and  3 
results  showed  a  100  percent  chance  that  the  results  are  from  three  separate  models. 

Note  that  the  desktop  simulation  model  produced  very  different  results  on  the 
LAMARS  (Table  6  and  Figure  21)  as  compared  to  the  VISTA  (Table  8  and  Figure  29). 
One  possible  reason  for  this  change,  compared  to  the  Level  1,  2,  and  3  models,  is  that  the 
desktop  model  had  an  active  pitch  rate  feedback  control  loop.  Or  one  of  the 
implementations  could  have  been  different  than  the  other  one.  Because  of  this 
discrepancy,  the  desktop  model  data  was  not  compared  directly  to  the  other  models,  but 
its  data  was  used  in  the  total  correlations  below. 
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VISTA  CHR-BAT  Correlation 


Figure  30  -  VISTA  CHR-BAT  Correlation 


Pilot  assigned  Cooper-Harper  ratings  were  plotted  against  the  subsequent  BAT 
task  simulation  run  time  until  boundary  impact.  As  can  be  seen  in  Figure  30,  the  data  is 
fairly  scattered.  If  the  test  subjects’  data  is  pared  down  to  only  the  test  subjects  with  pilot 
ratings,  the  results  are  more  coherent.  Figure  3 1  shows  the  same  data  with  only  pilot 
subjects  included.  One  data  point  was  also  removed  due  to  a  combination  of  factors  that 
were  deemed  to  have  caused  an  early  boundary  impact  by  the  test  subject:  clouds  in  the 
working  area,  improperly  adjusted  stick  arm-rest,  and  mannequin  effect. 
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VISTA  CHR-BAT  Correlation 


Figure  31  -  VISTA  CHR-BAT  Correlation  -  Pilot  Only  Test  Subjects 


Conclusions  were  then  made  to  predict  Cooper-Harper  ratings.  For  example, 
from  Figure  31,  it  can  be  concluded  that  should  a  pilot  fly  the  model  tracking  task  for  450 
seconds,  he  or  she  will  also  probably  rate  the  aircraft  between  4  and  7  on  the  Cooper- 
Harper  scale  for  that  same  task. 

One  additional  way  to  look  at  the  similarities  between  the  BAT  data  and  the  CHR 
data  is  to  put  the  BAT  data  into  a  1  Dimensional  chart,  similar  to  a  histogram  of  Cooper- 
Harper  ratings. 
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Linear  scale  comparison  of  BAT  times 
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Figure  32  -  VISTA  CHR-BAT  Comparison  -  Pilot  Only  Test  Subjects 


Figure  32  contains  the  same  data  points  as  Figure  31,  and  reveals  a  marked 
similarity  between  the  location  and  distribution  of  the  BAT  data  and  the  CHR  data.  This 
positive  correlation  and  similarities  between  BAT  boundary  impact  times  and  CHR 
ratings  shows  that  boundary  avoidance  tracking  tasks  can  be  used  to  compare  different 
aircraft  pitch  control  systems  and  achieve  an  accurate  measure  of  actual  aircraft  handling 
qualities. 
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5J.5.2  Mean  Tracking  Error  Analysis 

The  second  method  of  determining  aircraft  handling  qualities  was  based  on  mean 
tracking  error.  This  method  was  more  effective  in  detennining  boundary  avoidance,  and 
also  held  some  utility  in  determining  aircraft  handling  qualities.  A  typical  boundary 
tracking  task  can  be  seen  below  in  Figure  33.  The  figure  shows  a  plot  of  tracking  error 
and  the  instantaneous  displayed  boundaries  over  the  length  of  the  task  run.  Also  shown 
are  the  pilot’s  longitudinal  stick  inputs  on  the  same  time  scale.  The  run  was  tenninated 
when  the  tracking  error  equaled  the  current  boundary  value,  or  from  the  EP’s  FIUD  view, 
when  the  flight  path  marker  crossed  the  displayed  boundary. 


Pilot  7,  Level  1  model,  Boundary  Tracking  Task 


Figure  33  -  Tracking  Error,  Boundaries  and  Stick  Inputs  of  Typical  Boundary 

Tracking  Task 
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A  closer  look  at  the  tracking  error  throughout  the  run  shows  little  variability  in  the 
peak  errors  as  the  boundaries  impose  on  the  task.  However,  a  plot  of  the  average  tracking 
error  for  the  60  seconds  each  discreet  boundary  value  was  displayed  illustrates  that  the 
boundaries  have  a  definite  impact  on  pilot  performance.  As  shown  in  Figure  34,  as  the 
boundaries  got  closer  to  the  tracking  task  target,  the  pilot’s  average  error  decreased  to  a 
minimum  value.  After  this  point,  the  pilot  was  no  longer  able  to  increase  performance 
and  further  shrinking  of  the  boundaries  disrupted  the  target  track,  causing  an  increase  in 
average  tracking  error.  The  result  is  that  for  this  pilot  flying  this  task  on  this  ‘aircraft’, 
the  minimum  average  tracking  error,  or  best  performance  has  been  found.  Also,  a 
boundary  value  that  will  produce  the  best  performance  has  been  identified,  and  can  be 
used  in  subsequent  tests  to  drive  the  pilot  to  maximum  performance. 


Pilot  7  -  Level  1  model 


Figure  34  -  Average  Track  Error  per  Boundary  Step  Size 
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To  take  this  analysis  method  one  step  farther,  the  minimum  average  tracking  error 
for  all  test  subjects  in  each  boundary  step  size  on  each  model  was  compiled,  and  the 
minimum  values  for  each  model  and  boundary  step  size  were  plotted  in  Figure  35.  It  is 
readily  apparent  that  test  subjects  flying  the  Level  1  model  could  achieve  performance 
(defined  as  smaller  average  error)  that  was  equal  to  or  better  than  the  performance  on  the 
Level  2  model.  The  same  can  be  said  for  test  subjects  flying  the  Level  2  model  versus 
the  Level  3  model. 


Figure  35  -  Minimum  Average  Track  Error  -  All  Pilots,  All  Test  Runs 


Another  thing  to  note  from  Figure  35  is  the  fact  that  the  decrease  in  boundary  size 
does  not  consistently  affect  pilot  performance  until  the  boundaries  are  inside  of  5.6 
degrees.  This  is  theorized  to  be  the  point  of  boundary  awareness:  the  maximum 
boundary  size  that  will  consistently  cause  an  increase  in  task  performance.  The  best 
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average  performance  does  not  decrease,  and  in  many  cases  it  increases  while  the 
boundaries  are  5.6  degrees  or  larger.  The  early  increase  in  minimum  average  error  with 
time  may  be  due  to  pilot  familiarity  with  the  aircraft  model  and  workload  management 
prior  to  boundary  awareness.  That  the  boundary  awareness  performance  improvement 
seems  to  be  consistent  across  all  three  models  is  significant.  This  implies  that  the  setup  of 
the  tracking  task  and  boundary  relationships  are  factors  that  contribute  to  when  boundary 
awareness  occurs.  Or,  even  more  significantly,  that  all  pilots  may  have  the  same  limit  for 
boundary  awareness. 


Figure  36  -  Mean  of  Individual  Pilot’s  Average  Tracking  Errors 

Figure  36  shows  the  mean  of  the  average  track  errors  for  each  boundary  value 
from  all  test  subjects  and  all  test  runs.  The  same  boundary  awareness  break  point  at  5.6 
degrees  boundary  value  can  also  be  observed  in  this  figure.  It  is  interesting  to  note  that 
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the  mean  of  the  average  tracking  errors  for  the  Level  1  and  Level  2  models  are  within  3 
percent  for  boundary  values  of  1.33  and  1.78  degrees.  However,  none  of  the  test  pilots 
was  able  avoid  boundary  impact  on  the  Level  2  model  long  enough  to  achieve  the  1 
degree  boundary  value,  while  three  of  the  test  pilots  passed  540  seconds  on  the  Level  1 
model  test  run  and  flew  in  the  1  degree  boundary  value.  Further  analysis  and/or  testing 
of  these  two  pitch  control  models  at  small  boundary  values  is  warranted  to  detennine  why 
the  average  tracking  perfonnance  is  similar  only  at  these  small  boundary  values,  even 
though  absolute  minimum  tracking  error  was  different 

One  additional  comparison  can  be  shown  between  average  tracking  error  and 
boundary  size.  Figure  37  shows  the  mean  flight  path  error  in  the  5  seconds  prior  to 
boundary  impact  plotted  against  the  boundary  size  at  boundary  impact. 
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Figure  37  -  Mean  Track  Error  vs.  Boundary  Size  at  Impact 

The  grouping  of  the  boundary  impact  times  of  the  different  models  is  very 
revealing  as  it  shows  that  just  about  any  pilot,  regardless  of  experience  level,  could 
achieve  very  good  performance  on  the  Level  1  pitch  control  model.  However,  the  wide 
spread  of  the  Level  3  impact  times  shows  that  pilot  skill  level  and  experience  play  a  large 
role  in  detennining  the  best  performance  achievable  on  a  pitch  control  model  with  poor 
handling  qualities.  In  this  case,  the  magnitude  of  the  standard  deviation  of  the  impact 
times  for  each  model  seems  to  correlate  with  the  Cooper-Harper  Ratings,  as  seen  in  Table 
8  and  Figure  29. 
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6.  Conclusions  and  Recommendations 


6.1  Conclusions  of  Research 

Handling  qualities  research  and  testing  is  arguably  the  most  important  aspect  of  manned 
flight  test.  Verifying  that  the  response  of  the  pilot-aircraft  system  matches  the 
expectations  and  requirements  is  the  ultimate  goal  of  any  flight  test  program. 

Historically,  the  flight  test  community  has  relied  on  two  forms  of  data  to  evaluate 
handling  qualities:  pilot  comments  and  Cooper-Harper  ratings.  This  thesis  proposes  a 
new  set  of  techniques  and  data  to  evaluate  aircraft  handling  qualities. 

The  goal  of  this  research  was  to  determine  if  a  bounded  tracking  task  could  be 
used  to  produce  numerical  aircraft  handling  qualities  data.  To  accomplish  this  goal  a 
nonlinear  6  DOF  F-16  aircraft  model  and  bounded  flight  path  angle  (y)  tracking  task  were 
developed  on  a  desktop  computer.  Test  subjects  flew  the  model  and  tracking  task  with 
variable  flight  path  angle  boundaries  and  3  different  stabilator  rate  limits.  The  test 
subjects  then  flew  the  tracking  task  with  the  same  boundaries  and  rate  limits  on  the 
AFRIT  s  Infinity  Cube  simulator.  Stabilator  rate  limiting  was  discarded  as  a  method  to 
produce  different  handling  qualities,  and  further  investigation  focused  on  the  dissimilar 
results  obtained  in  the  different  simulators.  The  results  were  compared  to  determine  if 
the  different  control  schemes  present  in  the  two  simulators  produced  different  tracking 
performance.  Data  analysis  focused  on  the  average  tracking  error  per  boundary  size  as  a 
measure  of  task  performance,  as  well  as  the  length  of  time  the  simulation  was  flown  prior 
to  impacting  a  boundary. 

Research  was  continued  as  part  of  the  USAF  TPS  TMP  ‘BAT  DART’,  at 
Edwards  AFB,  CA.  Pitch  control  models  representative  of  Cooper-Harper  levels  1,  2, 
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and  3  were  developed  by  the  Calspan  Corporation  for  integration  into  the  LAMARS  and 
the  VISTA.  Cooper-Harper  performance  rating  definitions  were  developed  for  the 
tracking  task,  and  an  additional  7  test  subjects  flew  the  tracking  task  on  the  desktop 
simulator,  the  LAMARS,  and  the  VISTA.  Each  test  subject  flew  the  tracking  profile 
twice:  once  with  no  boundaries  to  produce  a  baseline  CHR,  and  once  with  boundaries  to 
collect  BAT  data. 

Differences  in  simulation  hardware,  software,  and  control  inceptors  called  into 
question  the  validity  of  the  desktop  simulation  and  LAMARS  data  in  comparison  with 
actual  flight  test  data.  Overlaps  and  shifts  in  the  range  of  model  results  in  the  LAMARS 
data  were  inconsistent  with  the  data  produced  during  flight  test  in  the  VISTA.  However, 
the  primary  goal  of  the  LAMARS  visit  was  to  prepare  for  the  flight  test  portion  of  the 
research,  and  this  was  accomplished  to  the  satisfaction  of  the  test  team. 

Analysis  of  the  VISTA  flight  test  data  produced  a  correlation  between  total 
simulation  run  time  prior  to  impacting  a  boundary,  and  Cooper-Harper  rating  for  each 
model.  However,  it  should  be  noted  that  this  correlation  is  only  valid  for  the  tracking 
task  and  boundaries  used  in  this  research.  Data  supporting  the  existence  of  boundary 
awareness  was  obtained  by  examining  the  average  flight  path  track  error  for  all  test 
subjects.  Prior  to  boundary  awareness,  a  decrease  in  boundary  size  did  not  consistently 
produce  an  increase  in  perfonnance;  whereas  immediately  after  boundary  awareness 
occurred,  nearly  all  pilots  on  all  models  improved  their  perfonnance. 

The  boundary  avoidance  tracking  (BAT)  flight  test  techniques  developed  during 
this  thesis  and  the  ‘BAT  DART’  TMP  have  been  shown  to  provide  consistent  and 
relevant  handling  qualities  data,  with  the  added  benefit  of  being  subject  to  statistical 
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analysis.  Pilot  comments  will  always  remain  the  primary  method  of  evaluating  and 
correcting  aircraft  handling  quality  deficiencies,  but  BAT  data  could  conceivably 
augment  or  even  replace  the  Cooper-Harper  rating  as  a  reference  data  set  for  comparing 
and  evaluating  aircraft  handling  qualities.  Further  research  and  flight  test  is  warranted  to 
possibly  generalize  the  proposed  flight  test  techniques  so  that  they  can  be  applied  on  a 
variety  of  aircraft  platforms. 


6.2  Lessons  Learned 

The  initial  tracking  task  was  designed  with  the  goal  of  eventually  implementing 
the  test  on  the  VISTA.  However,  little  knowledge  of  the  limitations  and  system 
requirements  of  the  VISTA  was  available  at  the  time.  A  thorough  understanding  of  the 
intended  platform  of  implementation  is  recommended  to  properly  design  a  flight  test  task. 

Creation  of  the  three  pitch  control  models  was  requested  of  the  Calspan 
Corporation  for  two  reasons:  1)  Again,  unfamiliarity  with  the  VISTA  software  interface 
and  limitations.  2)  Lack  of  time  available  to  the  test  team  due  to  the  regular  syllabus 
requirements  in  the  middle  of  TPS.  However,  the  time  gained  by  having  someone 
outside  the  test  team  build  and  implement  the  pitch  control  models  was  lost  to 
requirements  creep  caused  by  miscommunication.  This  was  further  exasperated  by  one 
contractor  being  responsible  for  passing  required  models  and  software  to  another 
contractor,  and  the  miscommunications  that  developed  therein.  Having  a  single 
individual  or  team  that  has  ownership  and  responsibility  for  creation,  distribution,  and 
implementation  of  new  models  or  ideas  is  recommended  to  avoid  miscommunication  of 
vital  project  concepts. 
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An  initial  goal  for  the  LAMARS  testing  was  to  have  aerodynamic  models,  pitch 
control  model  implementation,  recorded  parameters,  and  testing  procedures  identical  to 
those  planned  for  the  VISTA.  However,  practical  implementation  considerations  as  well 
as  the  complete  difference  in  hardware  and  software  systems  between  the  two  different 
testing  platforms  made  this  goal  unobtainable.  However,  the  primary  goal  of  test 
procedure  and  test  team  practice  was  met,  and  the  testing  continued  as  planned.  Focusing 
on  the  minimum  data  necessary  to  support  the  research,  in  this  case  different  models 
flown  on  one  system  -  the  VISTA,  allowed  the  research  to  continue  to  a  valid  conclusion. 

Historically,  simulation  of  fly-by-wire  or  highly  augmented  aircraft  has  produced 
results  which  don’t  necessarily  match  initial  flight  test  responses.  A  cautious  and  well 
thought  out  approach  to  stick  sensitivity  in  initial  flight  testing  of  new  aircraft  designs  or 
models  will  assist  in  mitigating  the  effect  of  any  unexpected  aircraft  responses. 


6.3  Recommendations  for  Action  and  Future  Research 

The  general  properties  of  the  tracking  task  used  throughout  the  research  were 
taken  from  Mr.  Gray’s  initial  pilot  in-the-loop  simulation.  However,  the  visual 
representation  of  the  simulation  infonnation  and  boundaries  chosen  for  this  research 
allowed  more  precise  aircraft  control  and  tracking  precision.  Thus,  the  tracking  task  was 
not  optimally  designed  for  the  simulations  used.  The  result  was  that  during  a  large 
percentage  of  the  tracking  task  execution,  the  pilot  was  out  of  the  loop  with  the  aircraft, 
and  the  short  period  response  dominated  the  aircraft  dynamics.  A  frequency  analysis  of 
the  pitch  stick  inputs  was  attempted  on  the  VISTA  data,  but  the  results  were  inconclusive. 
Four  frequencies  dominated  the  data,  obtained  from  fast  Fourier  transfonns  (FFTs)  and 


69 


power  spectral  densities  (PSDs)  of  the  pilot  pitch  stick  inputs.  These  four  frequencies 
corresponded  to  the  approximate  short  period  of  the  pitch  model,  and  the  three  input 
frequencies  to  the  tracking  task.  At  smaller  boundary  values,  some  frequencies  slightly 
higher  than  the  aircraft  short  period  began  to  manifest,  but  the  data  was  too  noisy  to  make 
any  conclusions.  If  the  data  recording  frequency  were  increased,  and  the  pilot  in-the-loop 
time  maximized,  it  might  be  possible  to  observe  a  change  in  the  pilot’s  input  bandwidth 
in  the  frequency  domain  as  the  boundaries  decrease  in  size. 

In  any  future  BAT  research  that  will  attempt  to  analyze  pilot  inputs  in  the 
frequency  domain,  it  is  recommended  that  the  tracking  task  be  designed  to  keep  the  pilot 
in  the  aircraft  control  loop  for  as  large  a  percentage  of  time  as  possible. 

Due  to  the  fact  that  the  frequency  content  of  a  pilot’s  stick  inputs  could  occur  up 
to  approximately  20  Hz,  the  minimum  sample  rate  to  observe  this  data  should  be 
approximately  50  Hz.  However,  the  data  from  the  VISTA  was  recorded  at  66.66  Hz  and 
the  FFTs  and  PSDs  were  still  extremely  noisy.  An  increase  in  sample  rate  might  allow  a 
better  analysis  of  the  data  in  the  frequency  domain. 
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Appendix  A  -  Aerodynamic  State  Space  Models 


Unaugmented  F-16  Longitudinal  State  Space  Equations  (Witte,  2004): 
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Hanley  Case  “B”  State  Space  Equations  used  in  Infinity  Cube  (Witte,  2004): 
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Calspan  constructed  Cooper-Harper  Level  1,  2,  and  3  models: 


Level  1  A  matrix: 
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B.  C.  and  D  matrices  common  to  all  Calsnan  models  (Level  1.  2.  and  3] 
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Appendix  B  -  Pitch  Control  Models 
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Appendix  C  -  Simulation  Environments 
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Appendix  D  -  Recorded  Parameter  Lists 


Desktop  Simulation: 


airspeed 

(KCAS) 

pitch  attitude  (0) 

(degrees) 

flight  path  angle  (y) 

(degrees) 

rate  of  change  of  flight  path  angle  (y) 

(degrees/sec) 

stick  force 

(lbs) 

elevator  displacement 

(degrees) 

load  factor 

(g) 

target  altitude 

(feet) 

target  airspeed 

(KCAS) 

target  flight  path  angle  (ytgt) 

(gamma) 

boundary  value 

(degrees) 

flight  path  track  error  (y error) 

(degrees) 

rate  of  change  of  flight  path  track  error  ( y  en-or) 

(degrees/sec) 

Infinity  Cube  Simulation: 

flight  path  angle  (y) 

(degrees) 

rate  of  change  of  flight  path  angle  (y) 

(degrees/sec) 

stick  force 

(lbs) 

elevator  command  (TEU) 
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LAMARS: 


time 

(sec) 

pitch  stick  force 

(lbs) 

aileron  stick  force 

(lbs) 

elevator  command  (TEU) 

(degrees) 

elevator  position  (TEU) 

(degrees) 
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flight  path  angle  (y) 

(degrees) 

rate  of  change  of  flight  path  angle  (y) 
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(degrees) 

rate  of  change  of  flight  path  track  error  ( y  en-or) 
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barometric  pressure  altitude 

(feet) 
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(sec) 

boundary 

(degrees) 

flight  path  angle  (y) 

(degrees) 

target  flight  path  angle  (ytgt) 

(degrees) 

flight  path  angle  track  error  (y error) 

(degrees) 

target  HUD  display  elevation 

(degrees) 

calibrated  air  speed 

(KCAS) 
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(feet) 
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(#) 

flight  path  marker  elevation  (in  HUD) 

(degrees) 

forward  pitch  stick  command 

(lbs) 

forward  roll  stick  command 

(lbs) 

forward  rudder  pedal  command 

(lbs) 

mach  number 

(M) 

acceleration  in  y-axis 

(g) 

acceleration  in  z-axis 

(g) 

rate  of  change  of  velocity  in  x-axis 

(ft/sec2) 

angle  of  attack  (a) 

(degrees) 

bank  angle  ((j)) 

(degrees) 

roll  rate  (p) 

(deg/sec) 

rate  of  change  of  roll  rate  ( p  ) 

(deg/sec2) 
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rate  of  change  of  pitch  rate  ( q  ) 
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yaw  rate  (r) 

rate  of  change  of  yaw  rate  ( r  ) 
record  number 
left  flaperon  position 
left  horizontal  tail  position 
left  leading  edge  flap  position 
right  flaperon  position 
right  horizontal  tail  position 
right  leading  edge  flap  position 
rudder  position 
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(degrees) 
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Appendix  E  -  Test  Subject  Data 
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Appendix  F  -  Rating  Scales 
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Appendix  G  -  Statistical  Analysis  of  VISTA  Data 
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